Information flow in a network model and the law of diminishing marginal returns 
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We analyze a simple dynamical network model which describes the limited capacity of nodes 
to process the input information. For a suitable choice of the parameters, the information flow 
pattern is characterized by exponential distribution of the incoming information and a fat-tailed 
distribution of the outgoing information, as a signature of the law of diminishing marginal returns. 
Similar behavior is observed in another network model, describing in a different fashion the law of 
diminishing marginal returns. The analysis of a real EEG data-set shows that similar phenomena 
may be relevant for brain signals. 
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Most social, biological, and technological systems can 
be modeled as complex networks, and display substantial 
non-trivial topological features ^j, i2l- Moreover, time se- 
ries of simultaneously recorded variables are available in 
many fields of science; the inference of the underlying net- 
work structure, from these time series, is an important 
problem that received great attention in the last years. A 
method based on chaotic synchronization has been pro- 
posed in Q , a method based on model identification has 
been described in Use of a phase slope index to de- 
tect directionalities of interactions has been proposed in 
i- 

The inference of dynamical networks is also related 
to the estimation, from data, of the fiow of information 
between variables, as measured by the transfer entropy 
@i 0- Wiener and Granger ^9] formalized the no- 
tion that, if the prediction of one time series could be 
improved by incorporating the knowledge of past values 
of a second one, then the latter is said to have a causal 
infiuence on the former. Initially developed for economet- 
ric applications. Granger causality has gained popularity 
also among physicists (see, e.g., Ifl- 12, 14- 16|) and even- 
tually became one of the methods of choice to study brain 
connectivity in neuroscience [l3|- Multivariate Granger 
causality may be used to infer the structure of dynamical 
networks from data as described in [l3|. It has been re- 
cently shown that for Gaussian variables Granger causal- 
ity and transfer entropy are equivalent [Toj . and this 
framework has also been generalized to other probabil- 
ity densities [13] ■ Hence a weighted network obtained by 
Granger causality analysis can be given an interpretation 
in terms of flow of information between different compo- 
nents of a system. This way to look at information flow 
is particularly relevant for neuroscience, where it is cru- 
cial to shed light on the communication among neuronal 
populations, which is the mechanism underlying the in- 
formation processing in the brain f2ol|. Furthermore, re- 



cent studies have investigated the economics implications 
of several network types mapping brain function 2l|, [22 1 . 

In many situations it can be expected that each node 
of the network may handle a limited amount of informa- 
tion. This structural constraint suggests that informa- 
tion flow networks should exhibit some topological evi- 
dences of the law of diminishing marginal returns [2^ . 
a fundamental principle of economics which states that 
when the amount of a variable resource is increased, while 
other resources are kept fixed, the resulting change in the 
output will eventually diminish [2^ . The purpose of 
this work is to introduce a simple dynamical network 
model where the topology of connections, assumed to be 
undirected, gives rise to a peculiar pattern of the informa- 
tion flow between nodes: a fat tailed distribution of the 
outgoing information flows while the average incoming 
information fiow does not depend on the connectivity of 
the node. In the proposed model the units, at the nodes 
the network, are characterized by a transfer function that 
allows them to process just a limited amount of the in- 
coming information. We show that a similar behavior is 
observed also in another network model, which describes 
in a different fashion the law of diminishing marginal re- 
turns; we then show that this relevant topological feature 
is found as well in real neural data. 

Our model is as follows. Given an undirected net- 
work of n nodes and symmetric connectivity matrix 
Aij e {0,1}, to each node we associate a real variable 
Xi whose evolution, at discrete times, is given by: 



,{t+l) = F\ Y^^v^At) I +<Jm, (1) 

where ^ are unit variance Gaussian noise terms, whose 
strength is controlled by cr; F is a transfer function chosen 
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as follows: 



F{a) = aa \a\ < 
F{a) ^ aO a> 
F{a) = ~a6 a < - 



(2) 



where 6* is a threshold value. This transfer function is 
chosen to mimic the fact that each unit is capable to 
handle a limited amount of information. For large 9 our 
model becomes a linear map. At intermediate values of 
9, the nonlinearity connected to the threshold will affect 
mainly the mostly connected nodes (hubs): the input 
AijXj to nodes with low connectivity will remain typi- 
cally sub-threshold in this case. We consider hierarchical 
networks generated by preferential attachment mecha- 
nism [26^ . From numerical simulations of eqs. ([T]), we 
evaluate the linear causality pattern for this system as 
the threshold is varied. We verify that, in spite of the 
threshold, variables are nearly Gaussian so that we may 
identify the causality with the information flow between 
variables fl9l|. We compute the incoming and outgoing 
information flow from and to each node, Ci„ and Co„t, 
summing respectively all the sources for a given target 
and all the targets for a given source. Then we eval- 
uate the standard deviation of the distributions of Ci„ 
and Co«t, varying the realization of the preferential at- 
tachment network and running eqs. (jlj for 10000 time 
points. 




FIG. 1: The ratio between the standard deviation of Cout and 
those of Cin, R, is plotted versus 9 for the three architec- 
tures of network: preferential attachment (PRE), determinis- 
tic scale free (SFN) and homogeneous (HOM). The parame- 
ters of the dynamical system are a = 0.1 and a = 0.1. Net- 
works built by preferential attachment are made of 30 nodes 
and 30 undirected links, while the deterministic scale free net- 
work of 27 nodes is considered. The homogeneous networks 
have 27 nodes, each connected to two other randomly chosen 
nodes. 

In figure dJ) we depict R, the ratio between the stan- 
dard deviation of Cout over those of c™, as a function of 



the 9. As the threshold is varied, we encounter a range of 
for which the distribution of Cm is much narrower than 
that of Cout- In the same figure we also depict the corre- 
sponding curve for deterministic scale free networks , 
which exhibits a similar peak, and for homogeneous ran- 
dom graphs (or Erdos-Renyi networks [1^), with R al- 
ways very close to one. The discrepancy between the dis- 
tributions of the incoming and outgoing causalities arises 
thus in hierarchical networks. We remark that, in order 
to quantify the difference between the distributions of Cm 
and Cout, here we use the ratio of standard deviations but 
qualitatively similar results would have been shown using 
other measures of discrepancy. 
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FIG. 2: Scatter plot in the plane Ci„ — Cout for undirected 
networks of 30 nodes and 30 links built by means of the pref- 
erential attachment mechanism. The parameters of the dy- 
namical system are a = 0.1 and a = 0.1. The points repre- 
sent the nodes of 100 realizations of preferential attachment 
networks, each with 10 simulations of eqs. ^ for 10000 time 
points. (Top-left) Scatter plot of the distribution for all nodes 
ai 9 = 0.001. (Top-right) Contour plot of the distribution for 
all nodes a.t 9 = 0.012. (Bottom-left) Scatter plot of the dis- 
tribution for all nodes a.t 9 = 0.1. (Bottom-right) The total 
causality (obtained summing over all pairs of nodes) is plot- 
ted versus 9; circles point to the values of 9 in the previous 
subfigures. 

In ^ we report the scatter plot in the plane Cm — Cout 
for preferential attachment networks and for some values 
of the threshold. The distributions of and Cout, with 9 
equal to 0.012 and corresponding to the peak of fig. ([T|), 
are depicted in fig. 1^: Cm appears to be exponentially 
distributed around a typical value, whilst Cout shows a 
fat tail. In other words, the power law connectivity, of 
the underlying network, influences just the distribution 
of outgoing causalities. 

In fig. Q we show the average value of Ci„ and Cout 
versus the connectivity k of the network node: Cout grows 
uniformly with fc, thus confirming that its fat tail is a 
consequence of the power law of the connectivity. On 
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FIG. 3; For the preferential attachment network, a.t 9 — 0.012, 
the distributions (by smoothing spline estimation) of Ci„ and 
Cout are depicted. Units on the vertical axis are arbitrary. 
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FIG. 5: For the model ((3| the ratio R, between the stan- 
dard deviation of Cout and those of Ci„, is depicted versus a. 
Preferential attachment networks, of n nodes and n links, are 
considered. 



the contrary appears to be almost constant: on av- 
erage the nodes receive the same amount of information, 
irrespectively of k, whilst the outgoing information from 
each node depends on the number of neighbors. 



network we associate the variable a;,- whose evolution is 
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FIG. 4: In the ensemble of preferential attachment networks 
of figure ([2]), at = 0.012, Ci„ and Cout are averaged over 
nodes with the same connectivity and plotted versus the con- 
nectivity. 



It is worth mentioning that since a precise estimation 
of the information flow is computationally expensive, our 
simulations are restricted to rather small networks; in 
particular the distribution of Cout appears to have a fat 
tail but, due to our limited data, we can not claim that 
it corresponds to a simple power-law. 

A fat tail in the distribution of Cout is observed also 
in the following model: to each node of an undirected 



Xi{t + 1) Xj(t){t) -t- tT^j(t) 



(3) 



where j{t) is a node chosen randomly, at each time i, 
in the set of the neighboring nodes of i. Equations ^ 
implement, in a different way from ([1}, the occurrence 
that nodes may handle a limited incoming information: 
at each time each node is influenced just by one other 
node. In figure ^ we depict i? as a function of a, for 
preferential attachment networks and for different size of 
the networks: the discrepancy between the distributions 
of Cin and Cout increases as the size of the network grows 
while keeping a fixed. 
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FIG. 6: For the EEG data the distributions of and Cout 
are depicted in a scatter plot (left) and in terms of their dis- 
tributions, obtained by smoothing spline estimation (right). 

Now we turn to consider real electroencephalogram 
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(EEG) data. We used recording obtained at rest from 
10 healthy subjects. During the experiment, which lasted 
for 15 min, the subjects were instructed to relax and keep 
their eyes closed. To avoid drowsiness, every minute the 
subjects were asked to open their eyes for 5 s. EEG was 
measured with a standard 10-20 system consisting of 19 
channels [sj. Data were analyzed usin g th e linked mas- 
toids reference, and are available from [29|. 

For each subject we considered several epochs of 4 sec- 
onds in which the subjects kept their eyes closed. For 
each epoch we computed multivariate Kernel Granger 
Causality [l^ using a linear kernel and a model order 
of 5, determined by leave-one-out cross-validation. We 
then pooled all the values for information flow towards 
and from any electrode and analyzed their distribution. 

In figure ([6]) we plot the incoming versus the outgoing 
values of the information flow, as well as the distributions 
of the two quantities: the incoming information seems ex- 
ponentially distributed whilst the outgoing information 
shows a fat tail. These results suggest that overall brain 
effective connectivity networks may also be considered 
in the light of the law of diminishing marginal returns. 
More interestingly, this pattern is reproduced locally but 
with a clear modulation: a topographic analysis has also 
been made considering the distribution of incoming and 
outgoing causalities at each electrode. In figure ([7]) we 
depict the map of parameter R, thus obtained, on the 
scalp; the law of diminishing marginal returns seems to 
affect mostly the temporal regions. This well defined 
pattern suggests a functional role for the distributions. 

It is worthwhile to mention that there are other mea- 
sures of directed brain connectivity, such as Directed 
Transfer Function, Partial Directed Coherence and Phase 
Slope Index, which are not directly related to informa- 
tion transfer. In particular, for DTF and PDC, defined in 
terms of physiological bands, the interpretation in terms 
of information flow is still debated [30|. On the other 
hand we verified that a significant discrepancy between 
the distributions of incoming and outgoing connectivi- 
ties holds also for these methods. Furthermore, bivariate 
measures do not display this asymmetry of the distribu- 
tions of Cin and Cout- this is not surprising, indeed it is 
well known that bivariate causality also account for indi- 
rect interactions, see e.g. Here we limited ourselves 
to linear information flow; the amount of nonlinear infor- 
mation transmission and its functional roles are not clear 
[ill] . It will be interesting to investigate these phenomena 
also in the nonlinear case. 

Summarizing, we have pointed out that the pattern of 
information flows among variables of a complex system 
is the result of the interplay between the topology of the 
underlying network and the capacity of nodes to handle 
the incoming information. By means of two simple toy 
models, we have shown that they may exhibit the law 
of diminishing marginal returns for a suitable choice of 
parameters. The analysis of a real EEG data-set has 



shown that similar patterns exist for brain signals and 
could have a specific functional role. 
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FIG. 7: The distribution on the scalp of R for EEG data. 
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